Correction of Accumulated Errors in Inertial Measurement Units Attached to a User

ABSTRACT

A system including: a plurality of sensor modules having inertial measurement units and attached to different parts of a user (e.g., head, hands, arms) to measure their orientations; a plurality of optical marks attached to the user; a camera attached to the user; and a computing device configured to correct an accumulated error by detecting the optical marks in an image generated by the camera and identifying mismatches in directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated based on the image and the corresponding directions of the optical marks (or the sensors, or parts of the users) as measured and/or calculated from the orientation measurements generated by the sensor modules.

RELATED APPLICATIONS

The present application relates to U.S. patent application Ser. No.15/847,669, filed Dec. 19, 2017 and entitled “Calibration of InertialMeasurement Units Attached to Arms of a User and to a Head MountedDevice,” U.S. patent application Ser. No. 15/817,646, filed Nov. 20,2017 and entitled “Calibration of Inertial Measurement Units Attached toArms of a User to Generate Inputs for Computer Systems,” U.S. patentapplication Ser. No. 15/813,813, filed Nov. 15, 2017 and entitled“Tracking Torso Orientation to Generate Inputs for Computer Systems,”U.S. patent application Ser. No. 15/792,255, filed Oct. 24, 2017 andentitled “Tracking Finger Movements to Generate Inputs for ComputerSystems”, U.S. patent application Ser. No. 15/787,555, filed Oct. 18,2017 and entitled “Tracking Arm Movements to Generate Inputs forComputer Systems”, and U.S. patent application Ser. No. 15/492,915,filed Apr. 20, 2017 and entitled “Devices for Controlling Computersbased on Motions and Positions of Hands.” The entire disclosures of theabove-referenced related applications are hereby incorporated herein byreference.

FIELD OF THE TECHNOLOGY

The embodiments disclosed herein relate to computer input devices ingeneral and more particularly but not limited to input devices forvirtual reality and/or augmented/mixed reality applications implementedusing computing devices, such as mobile phones, smart watches, similarmobile devices, and/or other devices.

BACKGROUND

U.S. Pat. App. Pub. No. 2014/0028547 discloses a user control devicehaving a combined inertial sensor to detect the movements of the devicefor pointing and selecting within a real or virtual three-dimensionalspace.

U.S. Pat. App. Pub. No. 2015/0277559 discloses a finger-ring-mountedtouchscreen having a wireless transceiver that wirelessly transmitscommands generated from events on the touchscreen.

U.S. Pat. App. Pub. No. 2015/0358543 discloses a motion capture devicethat has a plurality of inertial measurement units to measure the motionparameters of fingers and a palm of a user.

U.S. Pat. App. Pub. No. 2007/0050597 discloses a game controller havingan acceleration sensor and a gyro sensor. U.S. Pat. No. D772,986discloses the ornamental design for a wireless game controller.

Chinese Pat. App. Pub. No. 103226398 discloses data gloves that usemicro-inertial sensor network technologies, where each micro-inertialsensor is an attitude and heading reference system, having a tri-axialmicro-electromechanical system (MEMS) micro-gyroscope, a tri-axialmicro-acceleration sensor and a tri-axial geomagnetic sensor which arepackaged in a circuit board. U.S. Pat. App. Pub. No. 2014/0313022 andU.S. Pat. App. Pub. No. 2012/0025945 disclose other data gloves.

U.S. Pat. No. 9,600,925 and U.S. Pat. App. Pub. No. 2017/0053454disclose calibration techniques for a virtual reality system usingcalibration data from an imaging device and calibration data from aninertial measurement unit on a virtual reality headset.

U.S. Pat. App. Pub. No. 2016/0378204 discloses a system to track ahandheld device for a virtual reality environment using sensor data ofthe handheld device and the camera data of a head mounted display.

The disclosures of the above discussed patent documents are herebyincorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 illustrates a system to track movements of a user according toone embodiment.

FIGS. 2 and 3 illustrate examples user poses that may be combined tocalibrate inertial measurement units attached to the arms of a user.

FIG. 4 illustrates a system to control computer operations according toone embodiment.

FIG. 5 illustrates a pose of a user to calibrate inertial measurementunits attached to the arms of a user.

FIG. 6 illustrates a skeleton model for the calibration according to theuser pose of FIG. 5.

FIG. 7 illustrates the computation of rotational corrections calibratedaccording to the user pose of FIG. 5.

FIG. 8 shows a method to calibrate the orientation of an inertialmeasurement unit according to one embodiment.

FIG. 9 shows a detailed method to perform calibration according to oneembodiment.

FIG. 10 shows a skeleton model for the calibration of a head mounteddevice using the calibration pose of FIG. 5.

FIG. 11 illustrates the computation of a rotation for the calibration ofthe front facing direction of a head mounted device.

FIG. 12 illustrates the use of a camera for the calibration of the frontfacing direction of a head mounted device according to the user pose ofFIG. 5.

FIG. 13 illustrates the determination of a direction from an image forthe calibration of the front facing direction of a head mounted device.

FIG. 14 shows a method to calibrate the front facing direction of a headmounted device according to one embodiment.

FIG. 15 shows a detailed method to calibrate the front facing directionof a head mounted device according to one embodiment.

FIG. 16 illustrates the use of a head mounted camera for the directionmeasurements of optical marks on sensor modules.

FIG. 17 illustrates a scenario where the direction measurements ofoptical marks on sensor modules cannot be determined using a headmounted camera.

FIG. 18 illustrates a skeleton model for the correction of accumulatederrors in inertial measurement units according to one embodiment.

FIG. 19 illustrates corrections made according to the model of FIG. 18.

FIGS. 20 and 21 illustrate the computation of the orientation of aforearm based on the orientations of the hand and the upper armconnected by the forearm.

FIG. 22 shows a method to correct accumulated errors in an inertialmeasurement unit according to one embodiment.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

At least some embodiments disclosed herein allow the determination ofthe deviation of the measurement space of a sensor device from areference coordinate system using a convenient pose of a user wearingthe sensor device on arms of the user.

FIG. 1 illustrates a system to track movements of a user according toone embodiment.

In FIG. 1, a user wears a number of sensor devices (111, 113, 115, 117and 119). Each of the sensor devices (111-119) has an inertialmeasurement unit (IMU) to determine its orientation. Thus, the sensordevices (111-119) track the orientations of portions of user that aremovable relative to the torso (101) of the user and relative to eachother, such as the head (107), the upper arms (103 and 105), and thehands (106 and 108).

The sensor devices (111-119) communicate their movement/orientationmeasurements to a computing device (141), e.g., to control gesture inputand/or an avatar of the user in a virtual reality, mixed reality, oraugmented reality application in the computing device (141).

Typically, the measurements of the sensor devices (111-119) arecalibrated for alignment with a common reference system, such as acoordinate system (100).

In FIG. 1, the coordinate system (100) is aligned with a number ofdirections relative to the user when the user is in the reference poseillustrated in FIG. 1. The direction Z is parallel to the verticaldirection from the feet of the user to the head (107) of the user; thedirection Y is parallel to the sideway direction from one hand (106) ofthe user to another hand (108) of the user; and the direction X isparallel to the front-back direction relative to the torso (101) of theuser in the reference pose.

The directions X, Y and Z can be identified to the computing device(141) via combinations of reference poses, such as the poses illustratedin FIGS. 1-3.

When in the reference pose illustrated in FIG. 1, the arms (103 and 105)of the user extends horizontally in the sideways, in alignment with thedirection Y of the coordinate system (100).

When in the reference pose illustrated in FIG. 2, the arms (103 and 105)of the user extends vertically in alignment with the direction Z of thecoordinate system (100).

When in the reference pose illustrated in FIG. 3, the arms (103 and 105)of the user extends forwards and horizontally, in alignment with thedirection X of the coordinate system (100).

From the pose of FIG. 1 to the pose of FIG. 2, the arms (103 and 105) ofthe user rotate along the direction X. Thus, by comparing theorientations of the sensor devices (113-119) that rotate with the arms(103 and 105) along the direction X from the pose of FIG. 1 to the poseof FIG. 2, the computing device (141) computes the direction X in themeasurement spaces of the sensor devices (113-119).

From the pose of FIG. 2 to the pose of FIG. 3, the arms (103 and 105) ofthe user rotate along the direction Y. Thus, by comparing theorientations of the sensor devices (113-119) that rotate with the arms(103 and 105) along the direction Y from the pose of FIG. 2 to the poseof FIG. 3, the computing device (141) computes the direction Y in themeasurement spaces of the sensor devices (113-119).

From the pose of FIG. 3 to the pose of FIG. 1, the arms (103 and 105) ofthe user rotate along the direction Z. Thus, by comparing theorientations of the sensor devices (113-119) that rotate with the arms(103 and 105) along the direction Z from the pose of FIG. 3 to the poseof FIG. 1, the computing device (141) computes the direction Z in themeasurement spaces of the sensor devices (113-119).

Further, the direction Z can be computed as a direction orthogonal tothe directions X and Y in a three dimensional space; the direction Y canbe computed as a direction orthogonal to the directions X and Z in thethree dimensional space; and the direction X can be computed as adirection orthogonal to the directions Y and Z in the three dimensionalspace.

Thus, from the measured orientations of the sensor devices (113-119) inthe three reference poses illustrated in FIGS. 1-3, the computing devicecomputes the relative orientation change between the referencecoordinate system (100) and the measurement space of each of the sensordevices (113-119). Subsequently, the relative rotation can be used tocalibration the corresponding one of the sensor devices (113-119) suchthat the calibrated orientation measurements relative to the commonreference coordinate system (100).

The head sensor (111) can also be calibrated to produce measurementsrelative to the common reference coordinate system (100). For example,the average of the orientations of the head sensor (111) in thereference poses of FIGS. 1-3, or one of the orientations, can beconsidered as a reference orientation that is aligned with the commonreference coordinate system (100); and the subsequent measurements ofthe head sensor (111) can be calibrated as being relative to thereference orientation.

After the measurements of the sensor devices (111-119) are calibrated tomeasure relative to the common reference coordinate system (100), thehands, arms (105, 103), the head (107) and the torso (101) of the usermay move relative to each other and relative to the coordinate system(100). The measurements of the sensor devices (111-119) provideorientations of the hands (106 and 108), the upper arms (105, 103), andthe head (107) of the user relative to the common coordinate system(100).

In some implementations, each of the sensor devices (111-119)communicates its measurements directly to the computing device (141) ina way independent from the operations of other sensor devices.

Alternative, one of the sensor devices (111-119) may function as a baseunit that receives measurements from one or more other sensor devicesand transmit the bundled and/or combined measurements to the computingdevice (141).

Preferably, wireless connections made via a personal area wirelessnetwork (e.g., Bluetooth connections), or a local area wireless network(e.g., Wi-Fi connections) are used to facilitate the communication fromthe sensor devices (111-119) to the computing device (141).

Alternatively, wired connections can be are used to facilitate thecommunication among some of the sensor devices (111-119) and/or with thecomputing device (141).

For example, a hand module (117 or 119) attached to or held in acorresponding hand (106 or 108) of the user may receive the motionmeasurements of a corresponding arm module (115 or 113) and transmit themotion measurements of the corresponding hand (106 or 108) and thecorresponding upper arm (105 or 103) to the computing device (141).

For example, the hand modules (117 and 119) and the arm modules (115 and113) can be each respectively implemented via a base unit (or a gamecontroller) and an arm/shoulder module discussed in U.S. patentapplication Ser. No. 15/492,915, filed Apr. 20, 2017 and entitled“Devices for Controlling Computers based on Motions and Positions ofHands”, the entire disclosure of which application is herebyincorporated herein by reference.

In some implementations, the head module (111) is configured as a baseunit that receives the motion measurements from the hand modules (117and 119) and the arm modules (115 and 113) and bundles the measurementdata for transmission to the computing device (141). In some instances,the computing device (141) is implemented as part of the head module(111).

The head module (111) may further determine the orientation of the torso(101) from the orientation of the arm modules (115 and 113) and/or theorientation of the head module (111) and thus eliminates the need toattach a separate sensor device to the torso (101), as discussed in U.S.patent application Ser. No. 15/813,813, filed Nov. 15, 2017, thedisclosure of which is hereby incorporated herein by reference.

For the determination of the orientation of the torso (101), the handmodules (117 and 119) are optional in the system illustrated in FIG. 1.Further, in some instances the head module (111) is not used, in thetracking of the orientation of the torso (101) of the user.

Optionally, the computing device (141), or a hand module (e.g., 117),may combine the measurements of the hand module (e.g., 117) and themeasurements of the corresponding arm module (e.g., 115) to compute theorientation of the forearm connected between the hand (106) and theupper arm (105), in a way as disclosed in U.S. patent application Ser.No. 15/787,555, filed Oct. 18, 2017 and entitled “Tracking Arm Movementsto Generate Inputs for Computer Systems”, the entire disclosure of whichis hereby incorporated herein by reference.

FIG. 4 illustrates a system to control computer operations according toone embodiment. For example, the system of FIG. 4 can be implemented viaattaching the arm modules (115, 113) to the upper arms (105 and 103)respectively and optionally, the head module (111) to the head (107), ina way illustrated in FIGS. 1-3.

In FIG. 4, the head module (111) and the arm module (113) havemicro-electromechanical system (MEMS) inertial measurement units (IMUs)(121 and 131) that measure motion parameters and determine orientationsof the head (107) and the upper arm (103). Similarly, the hand module(119) may also have its IMU.

Each of the IMUs (131, 121) has a collection of sensor components thatenable the determination of the movement, position and/or orientation ofthe respective IMU along a number of axes. Examples of the componentsare: a MEMS accelerometer that measures the projection of acceleration(the difference between the true acceleration of an object and thegravitational acceleration); a MEMS gyroscope that measures angularvelocities; and a magnetometer that measures the magnitude and directionof a magnetic field at a certain point in space. In some embodiments,the IMUs use a combination of sensors in three and two axes (e.g.,without a magnetometer).

The computing device (141) has a motion processor (145), which includesa skeleton model (143) of the user (e.g., illustrated FIG. 6). Themotion processor (145) controls the movements of the corresponding partsof the skeleton model (143) according to the movements/orientations ofthe upper arms (103 and 105) measured by the arm modules (113 and 115),the movements/orientation of the head (107) measured by the head module(111), the movements/orientations of the hand (106 and 108) measured bythe hand modules (117 and 119), etc.

The skeleton model (143) is controlled by the motion processor (145) togenerate inputs for an application (147) running in the computing device(141). For example, the skeleton model (143) can be used to control themovement of an avatar/model of the arms (105 and 103), the hands (106and 108), the head (107), and the torso (101) of the user of thecomputing device (141) in a video game, a virtual reality, a mixedreality, or augmented reality, etc.

Preferably, the arm module (113) has a microcontroller (139) to processthe sensor signals from the IMU (131) of the arm module (113) and acommunication module (133) to transmit the motion/orientation parametersof the arm module (113) to the computing device (141). Similarly, thehead module (111) has a microcontroller (129) to process the sensorsignals from the IMU (121) of the head module (111) and a communicationmodule (123) to transmit the motion/orientation parameters of the headmodule (111) to the computing device (141).

Optionally, the arm module (113) and the head module (111) have LEDindicators (137 and 127) respectively to indicate the operating statusof the modules (113 and 111).

Optionally, the arm module (113) has a haptic actuator (138)respectively to provide haptic feedback to the user.

Optionally, the head module (111) has a display device (127) and/orbuttons and other input devices (125), such as a touch sensor, amicrophone, a camera, etc.

In some implementations, the head module (111) is replaced with a modulethat is similar to the arm module (113) and that is attached to the head(107) via a strap or is secured to a head mounted display device.

In some applications, the hand module (119) can be implemented with amodule that is similar to the arm module (113) and attached to the handvia holding or via a strap. Optionally, the hand module (119) hasbuttons and other input devices, such as a touch sensor, a joystick,etc.

For example, the handheld modules disclosed in U.S. patent applicationSer. No. 15/792,255, filed Oct. 24, 2017 and entitled “Tracking FingerMovements to Generate Inputs for Computer Systems”, U.S. patentapplication Ser. No. 15/787,555, filed Oct. 18, 2017 and entitled“Tracking Arm Movements to Generate Inputs for Computer Systems”, and/orU.S. patent application Ser. No. 15/492,915, filed Apr. 20, 2017 andentitled “Devices for Controlling Computers based on Motions andPositions of Hands” can be used to implement the hand modules (117 and119), the entire disclosures of which applications are herebyincorporated herein by reference.

FIG. 4 shows a hand module (119) and an arm module (113) as examples. Anapplication for the tracking of the orientation of the torso (101)typically uses two arm modules (113 and 115) as illustrated in FIG. 1.The head module (111) can be used optionally to further improve thetracking of the orientation of the torso (101). Hand modules (117 and119) can be further used to provide additional inputs and/or for theprediction/calculation of the orientations of the forearms of the user.

Typically, an IMU (e.g., 131 or 121) in a module (e.g., 113 or 111)generates acceleration data from accelerometers, angular velocity datafrom gyrometers/gyroscopes, and/or orientation data from magnetometers.The microcontrollers (139 and 129) perform preprocessing tasks, such asfiltering the sensor data (e.g., blocking sensors that are not used in aspecific application), applying calibration data (e.g., to correct theaverage accumulated error computed by the computing device (141)),transforming motion/position/orientation data in three axes into aquaternion, and packaging the preprocessed results into data packets(e.g., using a data compression technique) for transmitting to the hostcomputing device (141) with a reduced bandwidth requirement and/orcommunication time.

Each of the microcontrollers (129, 139) may include a memory storinginstructions controlling the operations of the respectivemicrocontroller (129 or 139) to perform primary processing of the sensordata from the IMU (121, 131) and control the operations of thecommunication module (123, 133), and/or other components, such as theLED indicator (137), the haptic actuator (138), buttons and other inputdevices (125), the display device (127), etc.

The computing device (141) may include one or more microprocessors and amemory storing instructions to implement the motion processor (145). Themotion processor (145) may also be implemented via hardware, such asApplication-Specific Integrated Circuit (ASIC) or Field-ProgrammableGate Array (FPGA).

In some instances, one of the modules (111, 113, 115, 117, and/or 119)is configured as a primary input device; and the other module isconfigured as a secondary input device that is connected to thecomputing device (141) via the primary input device. A secondary inputdevice may use the microprocessor of its connected primary input deviceto perform some of the preprocessing tasks. A module that communicatesdirectly to the computing device (141) is consider a primary inputdevice, even when the module does not have a secondary input device thatis connected to the computing device via the primary input device.

In some instances, the computing device (141) specifies the types ofinput data requested, and the conditions and/or frequency of the inputdata; and the modules (111, 113, 115, 117, and/or 119) report therequested input data under the conditions and/or according to thefrequency specified by the computing device (141). Different reportingfrequencies can be specified for different types of input data (e.g.,accelerometer measurements, gyroscope/gyrometer measurements,magnetometer measurements, position, orientation, velocity).

In general, the computing device (141) may be a data processing system,such as a mobile phone, a desktop computer, a laptop computer, a headmounted virtual reality display, a personal medial player, a tabletcomputer, etc.

After the initial calibration of the sensor devices (111-119) foralignment with the common coordinate system (100), the sensor devices(113-119) may move relative to the user. For example, the arm module(e.g., 115) may slip around the arm (105) of the user after a game invirtual reality or augmented reality implemented via the computingdevice (141). The sensor devices mounted on the arms of the user can beconveniently re-calibrated using a reference pose illustrated in FIG. 5.

FIG. 5 illustrates a pose of a user to calibrate inertial measurementunits attached to the arms of a user. For example, the calibration poseof FIG. 5 can be used for recalibration after the sensor devices(111-119) are initially calibrated according to the reference poses ofFIGS. 1-3 using the computing system illustrated in FIG. 4.

In FIG. 5, the user extends arms (103 and 105) and hands (106 and 108)straight in a horizontal plane XY. The hand modules (117 and 119) are incontact with each other. The arms (103 and 105) form small anglesrespectively with the front facing direction X. The left and right arms(103 and 105) and the left and right hands (106 and 108) are inpositions symmetric with respect to the center alone the front facingdirection X. The front facing direction of the torso (101) of the useraligns with the direction X. In some instances, the calibration pose ofFIG. 5 defines an updated coordinate system (100) that may have rotatedalong the vertical axis Z by an angle from the initial coordinate system(100) at the time of the calibration pose of FIG. 1.

The orientations of the arm modules (103 and 105) at the pose of FIG. 5can be used to calibrate the arm modules (103 and 105) for alignmentwith the common coordinate system (100), as further discussed below. Thecalibration corrects at least the twist or turn of the arm module (e.g.,115) around the arm (105) between the time of the initial calibration(e.g., calibrated according to the reference pose of FIG. 1) and thetime of recalibration using the pose of FIG. 5.

Further, the direction V_(A) (159) of the arm (105) in the updatedcoordinate system (100) aligned with the front facing direction of thetorso (101) at the calibration pose of FIG. 5 can be computed from theshoulder width of the user, the arm length of the user, and the width ofthe hand module (117) and used to calibrate the orienation of the armmodule (115) within the horizontal plane XY.

FIG. 6 illustrates a skeleton model for the calibration according to theuser pose of FIG. 5. For example, the skeleton model of FIG. 6 can beused to compute the direction V_(A) (159) of the arm (105) forcalibration in the horizontal plane XY.

In FIG. 6, the skeleton model has a known shoulder width L_(S) (155), aknown arm length_(S) (153), and a known width W (151) of the handhelddevice (117). Thus, the distance L (157) between the handheld device(117) and the torso (101), as well as the direction V_(A) (159) of thearm (105) relative to the updated coordinate system (100), can becomputed from the shoulder width L_(S) (155), the arm length_(S) (153),and the width of the handheld device (117).

The skeleton model of FIG. 6 illustrates an arm (e.g., 105) as one rigidunit. Alternatively, other skeleton models may include separate forearmsand upper arms that are connected via elbow joints and that may moverelative to each other. Further, a skeleton model may include the head(107) and/or other portions of the user.

FIG. 7 illustrates the computation of rotational corrections calibratedaccording to the user pose of FIG. 5. For example, the computation ofFIG. 7 can be performed in the computing system of FIG. 4 using thecalibration pose of FIG. 5 and the skeleton model of FIG. 6, after thesensor devices (111-119) are initially calibrated for alignment with acommon coordinate system (100) using the reference poses of FIGS. 1-3.

FIG. 7 illustrates a rotation γ (165) along the lengthwise direction ofthe arm (105) while the arm (105) is at the reference pose of FIG. 1 anda rotation τ within the horizontal plane XY along the direction Z.

The orientation change of the arm module (115) with respect to thecommon reference system (100) can be considered as the result of arotation γ (165) around the arm (105) while the arm (105) is pointingalong the direction Y and then a rotation (163) within the horizontalplane XY along the direction Z.

Thus, the rotation γ (165) can be computed by removing the rotation fromthe orientation change of the arm module (115) with respect to thecommon reference system (100).

For example, the rotational transformation of the arm module (115) asmeasured by the arm module (115) relative to the common reference system(100) can be applied, as a rotation α (161), to a unit direction that isinitially along the direction Y to obtain the lengthwise direct R (171)of the arm (105) at the time of the calibration pose of FIG. 5. Theprojection of the lengthwise direct R (171) at the time of thecalibration pose of FIG. 5, projected onto the horizontal plane XY alongthe direction Z, provides a direction R_(h) that has a rotation β (163)within the horizontal plane XY to the direction Y. The angle of therotation β (163) can be calculated as the rotation between the directionY and the direction R_(h). Reversing the rotation β (163) from therotational transformation of the arm module (115) from the referencecoordinate system (100) to the calibration pose of FIG. 5, as measuredby the arm module (115), provides the rotation γ (165).

When the user moves the arm (105) from the reference pose of FIG. 1 tothe calibration pose of FIG. 5, the arm (105) rotates along itslengthwise direction by a predetermined angle (e.g., 39 degrees). Thus,the difference between the calculated rotation γ (165) and thepredetermined angle (e.g., 39 degrees) provides a desired correction(e.g., for a result of the arm module (115) slipping around the arm(105) after the initial calibration).

After calibrating the measurement of the arm module (115) to account forthe slipping of the arm module (115) around the arm (105), it may bedesirable to further recalibrate to account for the rotation of thetorso (101) along the vertical direction, such that the front facingdirection X as calibrated is aligned with the front facing direction ofthe torso (101) at the time of the calibration pose of FIG. 5.

Since the arm (105) is expected to be in the direction V_(A) (159), therotation τ (167), rotating in the horizontal plane XY along thedirection Z from the direction R_(h) (173) to the direction V_(A) (159),can be applied to the orientation measurement of the arm module (115)such that the measurements of the arm module (115) is calibratedaccording to the new front facing direction identified by the torso(101) at the time of the calibration pose of FIG. 5.

The computation of the calibration parameters (e.g., γ (165) and τ(167)) can be performed in the computing device (141). Subsequently, thecomputing device (141) may apply the calibration parameters to themeasurement results received from the sensor device (e.g., 115).

Alternatively, the computing device (141) may transmit the calibrationparameters (e.g., γ (165) and τ (167)) to the sensor device (e.g., 115)that calibrates subsequent measurements relative to the commoncoordinate system (100) as defined by the calibration pose of FIG. 5.

In some implementations, the computing device (141) transmits theparameters used for the calibration calculation to the sensor device(e.g., 115), such as the direction V_(A) in the new coordinate system(100) aligned with the calibration pose of FIG. 5, and/or thepredetermined angle of rotation (e.g., 39 degrees) along the lengthwisedirection when the arm (e.g., 105) is rotated from the reference pose ofFIG. 1 to the calibration pose of FIG. 5. The sensor device (e.g., 115)calculates the calibration parameters (e.g., γ (165) and τ (167)) fromits current orientation measurement upon an indication from thecomputing device (141) that the user is in the calibration pose of FIG.5.

FIG. 8 shows a method to calibrate the orientation of an inertialmeasurement unit according to one embodiment. For example, the method ofFIG. 8 can be implemented in the computing system of FIG. 4 using acalibration pose of FIG. 5, the skeleton model of FIG. 6, and therotation relations illustrated in FIG. 7, after a sensor device iscalibrated using the reference poses of FIGS. 1-3.

The method of FIG. 8 includes: receiving (201) an measurement of theorientation of a sensor module (115) attached to an arm (105) of a userin a calibration pose (e.g., illustrated in FIG. 5) relative to areference pose (e.g., illustrated in FIG. 1); calculating (203) a firstrotation γ (165) of the sensor module (115) around a lengthwisedirection of the arm (105) from the reference pose (e.g., illustrated inFIG. 1) to the calibration pose (e.g., illustrated in FIG. 5);calculating (205) a second rotation τ (167) of the sensor module (115)around a vertical axis Z, from a projection R_(h) (173) of a lengthwisedirection R (171) of the arm (115) at the calibration pose (e.g.,illustrated in FIG. 5), projected in a horizontal plane XY, to anexpected lengthwise direction V_(A) (159) of the arm (105) in an updatedreference pose (e.g., a pose obtained by rotating the arm (105) in thehorizontal plane XY to the side, in parallel with the torso (101), to bein a pose similar to that illustrated in FIG. 1, without rotating thetorso (101)); and calibrating (207) subsequent orientation measurementsof the sensor module (115) using the first rotation γ (165) and thesecond rotation τ (167).

From the time of initial calibration with respect to the reference pose(e.g., illustrated in FIG. 1) and the time of recalibration at thecalibration pose (e.g., illustrated in FIG. 5), the front facingdirection of the torso (101) may have shifted by rotating along thevertical axis Z. Further, the sensor module (115) attached to the arm(105) via an armband may have slipped around the arm (105) about thelengthwise direction of the arm (105). The recalibration realigns theorientation measurement of the sensor module (115) with an updatedreference coordinate system (101) that is aligned with respect to theupdate reference pose, without requiring the user to make the updatereference pose for the recalibration. The update reference posecorresponds to the pose obtained by the user rotating the arm (105) fromthe calibration pose illustrated in FIG. 5 to a position where thelengthwise direction is in parallel with the torso (101) and isperpendicular to the front facing direction of the torso (101). Therecalibration is performed relative to the update reference pose withoutrequiring the user to actually make the update reference pose.

When the arm (105) rotates from the reference pose (e.g., illustrated inFIG. 1) to the calibration pose (e.g., illustrated in FIG. 5), the arm(105) rotates about its lengthwise direction by a predetermined angle(e.g., 39 degrees). The difference between the first rotation γ (165)and the rotation of the predetermined angle (e.g., 39 degrees)represents the slip of the sensor module (115) around the arm (105) fromthe time of the calibration according to the reference pose (e.g.,illustrated in FIG. 1) and the time of the recalibration using thecalibration pose (e.g., illustrated in FIG. 5). The slip can be removedfrom subsequent measurements. The second rotation τ (167) can be appliedto the subsequent measurements so that the subsequent measurements arecalibrated according to the front facing direction of the torso (101) atthe time of the calibration pose (e.g., illustrated in FIG. 5).

FIG. 9 shows a detailed method to perform calibration according to oneembodiment. For example, the method of FIG. 9 can be implemented in thecomputing system of FIG. 4 using a calibration pose of FIG. 5, theskeleton model of FIG. 6 and the rotation relations illustrated in FIG.7, after a sensor device is calibrated using the reference poses ofFIGS. 1-3.

The method of FIG. 9 includes: rotating (221) a lengthwise direction(e.g., parallel to the direction Y in FIG. 1 or 7) of an arm (105) froma reference pose (e.g., illustrated in FIG. 1) to a calibration pose(e.g., illustrated in FIG. 5) according to an orientation measurementfrom a sensor device (115) attached to the arm (105). The arm (105) isin a horizontal plane XY both in the reference pose (e.g., illustratedin FIG. 1) and in the calibration pose (e.g., illustrated in FIG. 5).

After the rotation α (161) according to the orientation measurement, thelengthwise direction of the arm (105) arrives at the direction R (171)illustrated in FIG. 7. The method of FIG. 9 further includes: projecting(223) the lengthwise direction R (171) onto the horizontal plane XY toobtain the direction R_(h) (173) illustrated in FIG. 7; and calculating(225) a rotation β (163), from the lengthwise direction (e.g., parallelto the direction Y in FIG. 1 or 7) of the arm (105) at the referencepose (illustrated in FIG. 1), to the lengthwise direction R_(h) (173) asprojected onto the horizontal plane XY.

The method of FIG. 9 further includes: removing (227) the rotation β(163) from an orientation change of the sensor device (115) from thereference pose (e.g., illustrated in FIG. 1) to the calibration pose(e.g., illustrated in FIG. 5) to obtain a rotation γ (165) along thelengthwise direction of the arm (105); and deducting (229), from therotation γ (165) along the lengthwise direction of the arm (105), apredetermined rotation of the arm (105) about its lengthwise directionfrom the reference pose (e.g., illustrated in FIG. 1) to the calibrationpose (e.g., illustrated in FIG. 5) to obtain a twist of the sensordevice (115) around the arm (e.g., from the time of the initialcalibration according to the reference pose to the time of thecalibration pose).

Optionally, the method of FIG. 9 further includes: calculating (231) arotation τ (167), from the lengthwise direction R_(h) (173) as projectedonto the horizontal plane XY, to a predetermined direction V_(A) (159)of the arm (105) in an updated reference pose (e.g., corresponding tothe calibration pose of FIG. 5, by changing to a pose similar to thecalibration pose of FIG. 1 through rotating the arms (103 and 105)without moving the torso (101)) to determine a rotation τ (167) of afront facing direction Y of the user from the time of the reference pose(illustrated in FIG. 1) to the time of the calibration pose (illustratedin FIG. 5).

The method of FIG. 9 further includes calibrating (233) subsequentorientation measurements of the sensor device (115) relative to theupdated reference pose by removing the twist and optionally applying therotation τ (167) of the front facing direction of the user.

The methods of FIGS. 8 and 9 can be implemented in the computing device(141) separate from the arm module (115), or implemented in the armmodule (115). In some instances, the calibration parameters arecalculated by the computing device (141) can be communicated to the armmodule (115) for combination with subsequent measurements. In otherimplementations, a handheld device functions as a primary input devicethat calculates the calibration parameters and/or performs thecalibration for the arm module (115) that functions as a secondary inputdevice.

When the user is in the calibration pose illustrated in FIG. 5, it isalso desirable to calibrate the orientation measurements of the headmodule (111).

FIG. 10 shows a skeleton model for the calibration of a head mounteddevice using the calibration pose of FIG. 5.

In FIG. 10, the arms (e.g., 103) of the user are in a horizontal planeXY. The head module (111) is above the horizontal plane containing thearms (e.g., 103) of the user and may or may not be aligned with thefront facing direction X in general.

In one method for the calibration of the orientation measurements of thehead module (111), a user is required to position the head module (111)that is mounted on the head (107) of the user such that the front facingdirection of the head mounted device (111) is parallel with or inalignment with the global front facing direction X as defined by thedirections of the arms (e.g., 103 and 105, as illustrated in FIGS. 5 and6).

For example, the user may be instructed to look straight aheadhorizontally in a horizontal plane such that the front facing directionof the head mounted device (111) is the same as the global front facingdirection X at the calibration pose of FIG. 5. When the head module(111) is positioned in such a way at the calibration pose, theorientation measurement of the head mounted device (111) at thecalibration pose relative to the reference pose (e.g., illustrated inFIG. 1) can be used to recalibrate the subsequent orientationmeasurements of the had module (111) according to the global referencesystem XYZ at the calibration pose of FIG. 5. Thus, the head module(111) and the arm modules (113 and 115) are all calibrated according tothe same global reference system XYZ at the calibration pose of FIG. 5.

FIG. 11 illustrates the computation of a rotation for the calibration ofthe front facing direction of a head mounted device.

In FIG. 11, the front facing direction X at the global coordinate systemof the reference system is rotated (305) to the direction H (301)according to the orientation measurement of the head module (111) at thecalibration pose. Specifically, the orientation measurement of the headmodule (111) at the calibration pose relative to the reference posedefines a rotation α (305) that rotates the front facing direction X inthe reference pose of FIG. 1 to the front facing direction H (301) inthe calibration pose of FIG. 5.

To perform the calibration, the direction H (301) is projected on to thehorizontal plane XY to obtain the projected direction H_(h) (303).

From the projected direction H_(h) (303), a rotation β (307) along thevertical direction Z from the front facing direction X in the referencepose of FIG. 1 to the projected direction H_(h) (303) can be calculatedfor the calibration of subsequent orientation measurements of the headmodule (111). For example, when the rotation (307) is subtracted fromthe subsequent orientation measurements of the head module (111) forcalibration, the calibrated orientation measurement is relative to theglobal front facing direction X at the calibration pose.

When the head module (111) includes a camera (or when the camera isattached to the head module (111), which may include a head mounteddisplay (HMD)), a direction X_(I) (323) of the head module (111) that isin alignment with the global front facing direction X at the calibrationpose can be calculated from an image captured using the camera, asfurther discussed below. When the direction X_(I) (323) can bedetermined using the camera, the user does not have to look straightahead in a horizontal plane to position the head module (111) in such away that the front facing direction of the head mounted device (111) isin alignment with the global front facing direction X. Since thedirection X_(I) (323) is in alignment with the global front facingdirection X, the rotational transformation of the head module (111) fromthe reference pose to the calibration pose can be applied to thedirection X_(I) (323) to obtain the direction H (301) that faces thefront direction. It's projection direction H_(h) (303) in the horizontalplane defines the rotation β (307) from the direction X. Thus, thecorrectional rotation β (307) can be determined with the use of thecamera without requiring the user to position the head module (111) in aprecise orientation where the front facing direction of the head module(111) is in alignment with the global front facing direction X at thecalibration pose (at least in the vertical plane that contains theglobal front facing direction X at the calibration pose).

FIG. 12 illustrates the use of a camera for the calibration of the frontfacing direction of a head mounted device according to the user pose ofFIG. 5.

In FIG. 12, each of the handheld devices (117 and 119) has an opticalmark (312 and 314) that can be captured and identified in an automatedway in an image generated by a camera (309) (e.g., a camera included inthe head module (111) or a camera attached to the head module (111),such as a head mounted display).

For example, the optical marks (312 and 314) may be implemented usingLight-Emitting Diode (LED) lights (e.g., visible lights or infraredlights detectable via the camera (309)) installed on the handhelddevices. The light-Emitting Diode (LED) lights are switched on when thehandheld devices (117 and 119) are in use and/or when the user is in thecalibration pose of FIG. 12 (or FIG. 5).

For example, the handheld devices (117 and 119) may include one or morebuttons which, when activated, indicate that the user is in thecalibration pose, which cause the light-Emitting Diode (LED) lights tobe on and cause the camera (309) to capture an image for calibration.

For example, being in the calibration pose of FIG. 12 (or FIG. 5) for atime period longer than a threshold can be identified as a gesture forcalibration. When the gesture is detected from the sensor modules (e.g.,113, 115, 117, and 119), the light-Emitting Diode (LED) lights areturned on to simplify the detection of their locations in the image(s)captured by the camera (309).

In some instances, the optical marks (312 and 314) may be inactiveelements, such as decorative marks made of paints, stickers, ageometrical feature of the house of the handheld devices (117 and 119).In some instances, a pattern analysis of the captured image of thehandheld devices (117 and 119) and/or the hands (106 and 108) of theuser is performed to recognize the handheld devices (117 and 119) and/orthe hands (106 and 108) and thus their locations in the image.

FIG. 12 illustrates the locations (313 and 315) of the optical marks(312 and 314) on the image (317) captured by the camera (309). A centerpoint O can be identified from the optical marks (312 and 314). Thedirection V_(O) (311) of the center point O relative to the camera (309)can be calculated from the field of view (319) of the camera, therelative position of the center point O on the image (317).

In general, the direction V_(O) is not in parallel with the front facingdirection of the camera (309) and is not in parallel with the frontfacing direction X of the global coordinate system XYZ (100) at thecalibration pose of FIG. 12.

FIG. 12 illustrates an orientation of the camera (309), where the frontfacing direction of the head module (111) and the camera (309) is inparallel with (aligned with) the front facing direction X of the globalcoordinate system XYZ (100) (e.g., when the user looks straight ahead inthe horizontal plane). In such a situation, the horizontal rotation ofthe front facing direction of the head module (111) and the camera (309)from the reference pose to the calibration pose identifies therotational correction for the head module (111). For example, theorientation measurement of the head module (111) at the calibration poseprovides a rotation transformation (305) from the reference pose to thecalibration pose. When the rotation transformation (305) is applied tothe front facing direction at the reference pose (e.g., illustrated inFIG. 1) to obtain the direction H (301) as illustrated in FIG. 11, itsprojection H_(h) (303) defines a rotate β (307) from the front facingdirection X at the calibration pose, which can be used to re-calibratethe subsequent orientation measurements relative to the front facingdirection X of the global coordinate system (100) at the calibrationpose of FIG. 12, as discussed above in connection with FIG. 11.

In general, the user may not look straight ahead in the horizontalplane. Thus, the center point of the optical marks (312 and 314)captured in the image (317) in general may not be at point O. Instead ofat the direction V_(O) (311), the center point of the optical marks (312and 314) captured in the image (317) in general is at a direction V_(I)(321) relative to the camera (309) (as illustrated in FIG. 13). Thedeviation of the direction V_(I) (321) from the direction V_(O) (311)can be used to identify a direction X_(I) (323) that is in alignmentwith the front facing direction X of the global coordinate system (100)at the calibration pose of FIG. 12. In general, the direction X_(I)(323) is not in alignment with the front facing direction of the camera(309). After the direction X_(I) (323) is determined from the image(317) captured by the camera (309), applying the rotation transformationbetween the reference pose and the calibration pose to the directionX_(I) (323) provides the direction H (301) illustrated in FIG. 11, therotation correction β (307) from the front facing direction X at thereference pose can be determined from its projection H_(h) (303) in thehorizontal plan XY, as illustrated in FIG. 11, without requiring theuser to look straight ahead in the horizontal plane.

FIG. 13 illustrates the determination of a direction X_(I) (323) from acamera image captured at a calibration pose of FIG. 12 or FIG. 5 for thecalibration of the front facing direction of a head mounted device(111).

In FIG. 13, when the front facing direction x of the camera (309) andthe head module (111) is aligned with the global front facing directionX (e.g., as in FIG. 12), the direction V_(O) (311) of the center O ofthe optical marks (312, 314) captured by the camera (309) in the image(317) has a rotational offset ω (322) from the front facing direction x(e.g., as in FIG. 12).

When the camera (309) is rotated to capture the center O of the opticalmarks (312, 314) at the direction V_(I) (321), the direction X_(I) (323)is in alignment with the global front facing direction X. As illustratedin FIG. 13, the direction X_(I) (323) has the same rotational offset ω(322/324) from the direction V_(I) (321). Thus, the direction X_(I)(323) can be calculated by applying the rotational offset ω (322/324) tothe direction V_(I) (321). Alternatively, the rotation (326) from thedirection V_(O) (311) to the direction V_(I) (321) can be applied (325)to the front facing direction x of the camera (309) to obtain thedirection X_(I) (323).

FIG. 13 illustrates the camera rotation (325 or 326) in the horizontalplane where the user at the calibration pose looks in the horizontalplane but does not look exactly straight ahead. In general, the user mayalso look down at the optical marks (312 and 314) and thus does not lookin the horizontal plane. In such a situation the camera rotation fromthe direction V_(O) to the direction V_(I) may include a rotation out ofthe horizontal plane. The determination of the direction X_(I) (323) canbe performed by applying the rotational offset ω (322) to the directionV_(I) (321), or applying the rotation (326) to the front facingdirection x, in the same way as illustrated in FIG. 13.

The offset ω (322) can be calculated from the coordinate offsets of thecenter point O in the local coordinate system of the camera (309) at atime the local coordinate system of the camera (309) is aligned with theglobal coordinate system XYZ (100) as illustrated in FIG. 12. Forexample, the coordinate offsets include the vertical offset between thecamera (309) and the horizontal plane that goes through the center O atthe calibration pose, the lateral offset between the camera (309) and avertical plane that goes through the center of the user at thecalibration pose and the center O, and the frontal offset between thecamera (309) and a vertical plane that is perpendicular to the globalfront facing direction X and goes through the center O.

Since the direction X_(I) (323) is aligned with the direction X of theglobal coordinate system XYZ (100) at the time of the calibration pose(e.g., as illustrated in FIG. 5 or 12), the orientation transformationmeasured at the calibration pose relative to the reference pose (e.g.,as illustrated in FIG. 1) can be applied to the direction X_(I) (323) atthe reference pose to obtain the direction H (301) illustrated in FIG.11. A correction according to the rotation β (307) between the directionX and the projection H_(h) (303) as illustrated in FIG. 11 can beapplied to the subsequent orientation measurements of the head module(111) such that the calibrated measurements are relative to the globalcoordinate system XYZ (100) defined at the time of the calibration pose(e.g., illustrated in FIG. 5 or 15).

FIG. 14 shows a method to calibrate the front facing direction of a headmounted device according to one embodiment. For example, the method ofFIG. 14 can be implemented in a system illustrated in FIG. 4, using acalibration pose illustrated in FIG. 5 or 12 relative to a referencepose illustrated in FIG. 1 and a skeleton model of FIG. 10, inaccordance with the rotational offset illustrated in FIG. 13.

The method of FIG. 14 includes: receiving (331) an measurement of theorientation of a sensor module (111) attached to a head mounted device(111) of a user in a calibration pose (e.g., illustrated in FIG. 5 orFIG. 12) relative to a reference pose (e.g., illustrated in FIG. 1);identifying (333) a first direction X_(I) (323) of the sensor module(111) in the reference pose (e.g., illustrated in FIG. 1) that isexpected to be rotated to a front facing direction X of the globalcoordinate system (100) at the calibration pose (e.g., illustrated inFIG. 5 or FIG. 12); applying (335) a first rotation a (305), identifiedby the measurement of the orientation of the sensor module (111), to thefirst direction X_(I) (323) to obtain a second direction H (301) of thesensor module (121) at the calibration pose (e.g., illustrated in FIG. 5or 12) relative to the reference pose (e.g., illustrated in FIG. 1);determining (337) a second rotation β (307) between the front facingdirection X at the reference pose (e.g., illustrated in FIG. 1) and aprojection H_(h) (303) of the second direction H (301) in a horizontalplane XY; and calibrating (339) subsequent orientation measurements ofthe sensor module (121) using the second rotation β (307).

When the user is instructed to look straight ahead in the horizontalplane at the calibration pose (e.g. as illustrated in FIG. 12), thefirst direction X_(I) (323) can be identified based on the front facingdirection X at the time of the calibration pose (e.g., illustrated inFIG. 1).

When the head mounted device (111) has a camera (309) as illustrated inFIG. 12, it is not necessary to instruct the user to look straight aheadin the horizontal plane at the calibration pose. For example, the usermay look down into the optical marks (312, 314) with or without arotation along the vertical axis Z. In such a configuration, the firstdirection X_(I) (323) can be computed from the image (317) of theoptical marks (312 and 314) captured by the camera (309) at the time ofthe calibration pose (e.g., illustrated in FIG. 12), using the relationsillustrated in FIG. 13.

FIG. 15 shows a detailed method to calibrate the front facing directionof a head mounted device according to one embodiment. For example, themethod of FIG. 15 can be implemented in a system illustrated in FIG. 4,using a calibration pose illustrated in FIG. 5 or 12 relative to areference pose illustrated in FIG. 1 and a skeleton model of FIG. 10, inaccordance with the rotational offset illustrated in FIG. 13.

The method of FIG. 15 includes: capturing (341), using a camera (309) ofa head mounted device (111), an image (317) of optical marks (312 and314) attached on handheld devices (117 and 119) that are in the hands ofa user while the user is in a calibration pose (e.g., illustrated inFIG. 12); obtaining (343), from the head mounted device (111), anorientation measurement of the head mounted device (111) at thecalibration pose (e.g., illustrated in FIG. 12) relative to a referencepose (e.g., illustrated in FIG. 12); identifying (345), from the image(317), a direction V_(I) (321) of a center O of the optical marksrelative to the head mounted device (111); applying (347) an offset co(324) to the direction V_(I) (321) to obtain a first direction X_(I)(323) of the head mounted device (111) at the reference pose (e.g.,illustrated in FIG. 12); computing (349) a second direction H (301) ofthe head mounted device (111) at the calibration pose (e.g., illustratedin FIG. 1) that is rotated by an angle α (305) from the first directionX_(I) (323) according to the orientation measurement; projecting (351)the second direction H (301) to a horizontal plane XY to obtain a thirddirection H_(h) (303); calculating (353) a rotation (307) from a frontfacing direction X of the user at the reference pose (e.g., illustratedin FIG. 1) to the third direction H_(h) (303) in the horizontal planeXY; and calibrating (355) subsequent orientation measurements of thehead mounted device (111) using the rotation β (307).

The present disclosure includes a system that has a sensor module (111)having an inertial measurement unit (121) and attached to a head (107)of a user, and a computing device (e.g., 141) coupled to the sensormodule (111). The computing device (141) may be separate from the sensormodule (111), as illustrated in FIG. 4, or be part of the sensor module(111). In some instances, the sensor module (111) is a head mounteddisplay device with a camera (309).

The sensor module (111) generates a measurement of an orientation of thehead (107) at a calibration pose (e.g., illustrated in FIG. 5 or 12)relative to a reference pose (e.g., illustrated in FIG. 1). Thecomputing device (141) is configured to: apply a first rotation (e.g., a(305)), defined by the measurement of the orientation of the head at thecalibration pose relative to the reference pose, to a first direction(e.g., X or X_(I)) relative to the head at the reference pose to obtaina second direction H (301) relative to the reference pose; project thesecond direction H (301) on to a horizontal plane XY to obtain aprojected direction H_(h) (303) within the horizontal plane XY relativeto the reference pose; compute a second rotation β (307) between theprojected direction H_(h) (303) and a predetermined direction X of thehead (107) relative to the reference pose; and calibrate subsequentorientation measurements of the sensor module (111) using the secondrotation β (307).

For example, at the calibration pose as illustrated in FIG. 5 or 12,arms (103 and 105) of the user are in a horizontal plane XY to formpredetermined angles with respect to a front facing direction X of atorso of the user at the calibration pose; and at the reference pose asillustrated in FIG. 1, the arms (103 and 105) are in the horizontalplane and are perpendicular to a front facing direction X of the torsoof the user at the reference pose. When the calibration is subsequentlyre-performed, the previously performed calibration pose becomes thereference pose for the subsequent calibration.

The user may be instructed to look straight ahead horizontally; and insuch a situation, the first direction (e.g., X or X_(I)) is a frontfacing direction X at the reference pose.

When the sensor module (111) is a head mounted device having a camera(309), it is not necessary for the user to look straight aheadhorizontally at the calibration pose. The first direction X_(I) (323)can be determined from an image (317) captured by the camera (309) atthe calibration pose.

For example, the image (317) captures at least one optical mark (e.g.,312, 314) positioned in front of the user at the calibration pose (e.g.,illustrated in FIG. 12); and the first direction X_(I) (323) isdetermined based on a direction V_(I) (321) of the at least one opticalmark (e.g., 312, 314) relative to the camera (309) at the calibrationpose.

As illustrated in FIG. 13, the computing device may apply apredetermined offset ω (322/324) to the direction V_(I) (321) of the atleast one optical mark (e.g. 312, 314) relative to the camera (309) toobtain the first direction X_(I) (323). The predetermined offset ω(322/324) is equal to an offset, at a time when the user looks straightahead horizontally, between a direction V_(O) (311) of the at least oneoptical mark relative to the camera (309) and a front facing direction Xor x.

Alternatively, as illustrated in FIG. 13, the computing device maycompute a rotation offset (326), from a direction V_(I) (321) of the atleast one optical mark (312 and 314) relative to the camera (309) at thecalibration pose, to a direction V_(O) (311) of the at least one opticalmark (312 and 314) relative to the camera (309) when a front facingdirection x of the sensor module is in parallel with a front facingdirection X of a torso (101) of the user at the calibration pose. Then,the first direction X_(I) (323) can be computed based on the rotationoffset (326) by applying the equal rotation offset (325) to the frontfacing direction x of the sensor module (111).

For example, two handheld devices (117 and 119) having inertialmeasurement units can be used to measure orientations of hands (106 and108) of the user. The at least one optical mark (312 and 314) can beconfigured on the two handheld devices (e.g., 117 and 119) as LEDlights, painted marks, structure features, etc. The calibration pose maybe detected based on one or more user input generated using the handhelddevices (117 and 119), such as button clicks, touch inputs, joystickinputs.

In some instances, two arm modules (113 and 115) having inertialmeasurement units (e.g., 131) are also used to measure orientations ofthe arms (103 and 105) of the user. The calibration pose can be detectedbased on a gesture determined using orientation measurements of the armmodules (103 and 105) and/or the handheld devices (117 and 119).

The sensor module (111) may include a communication device (123) for acommunication link with the computing device (141). The inertialmeasurement unit (121) of the sensor module (111) may include amicro-electromechanical system (MEMS) gyroscope, a magnetometer, and/ora MEMS accelerometer.

The inertial measurement units (e.g., 121, 131) in the sensor modules(e.g., 111 to 119) measure their orientations and/or positions throughintegration of rate measurements (e.g., acceleration, angular velocity)over a period of time starting from a calibration (e.g., performed usingposes illustrated in FIGS. 1-3, FIG. 5, or FIG. 12). Errors in the ratemeasurements accumulate over time through the integration during theperiod of time, which can result in an increasing drift of theorientation and/or position measurements from their accurate values.After a threshold period of time, the drift in orientation and/orposition measurements of the inertial measurement units (e.g., 121, 131)can be corrected via another calibration operation.

The frequency of the calibrations of the inertial measurement units(e.g., 121, 131) can be reduced by correcting the accumulated errorsbased on the directions of the optical marks (312 and 314) measuredusing the camera (309) during the application usage of the sensormodules (e.g., 111 to 119), such as during a session of virtual reality,mixed reality or augmented reality display. The directions of theoptical marks (312 and 314) as captured in the images generated by thecamera (309) can be compared with the corresponding directions computedfrom the orientation and/or position measurements of the inertialmeasurement units (e.g., 121, 131). The orientation and/or positionmeasurements of the inertial measurement units (e.g., 121, 131) can becorrected to remove or reduce the differences between the optical markdirections measured using the camera (309) and the respective directionsdetermined using the inertial measurement units (e.g., 121, 131).

FIG. 16 illustrates the use of a head mounted camera for the directionmeasurements of optical marks on sensor modules.

In FIG. 16, the user has sensor modules (111 to 119) attached to thehead (107), upper arms (103 and 105) and hands (106 and 108), in a waysimilar to the configuration illustrated in FIG. 1. At least some of thesensor modules (e.g., 117, 119, 113, and/or 115) have optical marks(e.g., 312, 314) in a way as illustrated in and discussed above inconnection with FIG. 12.

After the sensor modules (111 to 119) are calibrated through the use ofposes of FIGS. 1 to 3 and/or subsequently recalibrated using the pose ofFIG. 5 or 12, the sensor modules (111 to 119) may have accumulatederrors through integration over a period of time when some of theoptical marks are detectable in the image generated by the camera (309)of the head module (111). As a result of the accumulated errors, thedirections and/or the positions of the optical marks (312, 314) on thehand modules (117 and 119) as determined from the image may be differentfrom the directions and/or the positions of the optical marks (312, 314)(e.g., as illustrated in FIG. 18) tracked via the inertial measurementunits (e.g., 121, 131) of the sensor modules (111 to 119).

In FIG. 16, the optical marks (312 and 314) are in the field of view(319) of the camera (309) of the head module (111). Using the image(s)generated by the camera (309) and the orientation of the head module(111), the computing device (141) calculates one or more directions ofthe optical marks (312, 314) relative to the user in a global coordinatesystem (100). Separately, the computing device (141) calculates thecorresponding one or more directions of the optical marks (312, 314)relative to the user in the global coordinate system (100) according tothe orientations and/or positions of the arm modules (113 and 115)and/or the hand modules (117 and 119). The differences between theoptical mark directions computed from the image generated by the camera(309) and from the sensor modules (113, 115, 117 and/or 119) areindications of accumulated errors. The orientations and/or positions ofthe head module (111), the arm modules (113 and 115) and/or the handmodules (117 and 119) can be adjusted to reduce or eliminate thedifferences.

FIG. 16 illustrates a scenario where at least some of the optical marks(e.g., 312 and 314) are within the field of view (319) of the camera(309), which allows the computing device (141) to reduce or eliminatethe accumulated errors in the inertial measurement units (e.g., 121,131) in the sensor modules (e.g., 111 to 119) using the images generatedby the camera (309). However, when the optical marks (e.g., 312 and 314)are moved out of the field of view (319) (e.g., as illustrated in FIG.17), the errors in the inertial measurement units (e.g., 121, 131) willaccumulate over time.

FIG. 17 illustrates a scenario where the direction measurements ofoptical marks on sensor modules cannot be determined using a headmounted camera (309). If the optical marks (e.g., 312 and 314) are outof the field of view for a period of time that is longer than athreshold, the accumulated errors may be sufficiently large; and theuser may be prompted to enter a calibration pose as illustrated in FIG.5 or 12, or recalibration via poses illustrated in FIGS. 1 to 3, or moveat least some of the optical marks (e.g., 312 and 314) into the field ofview (319) (e.g., as illustrated in FIG. 16).

FIGS. 16 and 17 illustrate a configuration where the computing device(141) is separate from the sensor modules (111 to 119). In otherconfigurations, the computing device (141) is integrated within the headmodule (111), or one of the hand modules (117 and 119).

Optionally, the camera (309) implements a stereoscopic vision for thedetermination of a position of an optical mark (e.g., 312 or 314) in thefield of view (319), which allows the determination of not only thedirection of the optical mark (e.g., 312 or 314) relative to the camera(309) but also the distance of the optical mark (e.g., 312 or 314) tothe camera (309). Based such the measurement of the direction anddistance of the optical mark (e.g., 312 or 314), the direction of theoptical mark (e.g., 312 or 314) relative to other parts of the user,such as the shoulder joints (102 and 104) of the user, can also becalculated based on a skeleton model of the user. When the camera (309)does not have a stereoscopic vision; and the direction of optical mark(e.g., 312 or 314) relative to the camera (309) is used directly orindirectly in the reduction or elimination of accumulated errors, asfurther discussed below.

FIG. 18 illustrates a skeleton model for the correction of accumulatederrors in inertial measurement units according to one embodiment.

The skeleton model of FIG. 18 has the positions of the arms (103 and105) and the hand modules (117 and 119) tracked using the orientationmeasurements of at least the arm models (113 and 115). FIG. 18 alsoshows the positions (367 and 369) of the hand modules (117 and 119) asobserved by the camera (309).

When the camera (309) has a stereoscopic vision, the camera (309)determines not only the direction from the camera (309) to the position(367) of the hand module, as indicated by the optical mark (314)observed by the camera (309), but also the depth of the position (367)of the hand module (117) from the camera (309). The combination of thedirection and depth provides the location of the position (367) in thespace, which can be subsequently used to compute the direction T_(R)(361) from the shoulder joint (104) of the user. The arm module (115)provides the lengthwise direction V_(R)(371) of the right arm (105). Therotation θ_(R) (365) from the direction V_(R) (371), as measured by thearm module (115), to the direction T_(R) (361), as measured using thecamera (309), represents an accumulated error in the orientationmeasurement of the arm module (115). Therefore, the rotation θ_(R) (365)can be applied to the orientation measurement of the arm module (115) toremove or reduce the accumulated error.

Similarly, FIG. 18 illustrates the lengthwise direction V_(L) (373) ofthe left arm (103), as measured by the arm module (113), and thedirection T_(L) (363) of the hand module (119), as measured using thecamera (309). The rotation θ_(L) (366) between the direction V_(L) (373)and the direction T_(L) (363) can be applied to the orientationmeasurement of the arm module (113) to remove or reduce the accumulatederror.

In some implementations, the common mode of the rotations θ_(L) (366)and θ_(R) (365) are considered the accumulated error of the orientationof the head module (111) to which the camera (309) is attached; and thedifferences between the common mode and the rotations θ_(L) (366) andθ_(R) (365) respectively are used to correct the orientationmeasurements of the arms (103 and 105). For example, a common mode ofrotation of (θ_(L)−θ_(R))/2 can be applied to the orientationmeasurement of the head module (111); a rotation ofθ_(L)−(θ_(L)−θ_(R))/2 is applied to the orientation measurement of theleft arm module (113); and a rotation of θ_(R)−(θ_(R)−θ_(L))/2 isapplied to the orientation measurement of the right arm module (115).

When the camera (309) does not have a stereoscopic vision, the position(367) of the hand module (117) can be determined from the intersectionbetween the line of sight of the hand module (117) as observed by thecamera (309) and the circle (spherical surface) that is centered at theshoulder joint (104) and has a radius equal to the distance between thehand module (117) and the shoulder joint (104). Once the position (367)of the hand module (117) is determined in the space, the directionT_(R)(361) pointing from the shoulder joint (104) to the position (367)can be calculated for the removal or reduction of the accumulatederrors.

After the orientations of the arm modules (113 and 115) are correctedaccording to the observed positions (367 and 369) of the hand modules(117 and 119), as indicated by their optical markers (312 and 314)observed by the camera (309), the skeleton model of FIG. 18 is updatedto a state as illustrated in FIG. 19.

FIG. 19 illustrates corrections made according to the model of FIG. 18.In FIG. 19, the positions of the hand modules (117 and 119) asdetermined from the orientation measurements of the arm modules (113 and115) coincide with their positions observed by the camera (309).

In some instances, the skeleton model of FIG. 18 or 19 is used tocontrol a computer display, such as an avatar of the user in virtualreality, or a display of augmented reality or mixed reality. Aninstantaneous correction from the skeleton model of FIG. 18 to theskeleton model of FIG. 19 may create undesirable artifacts in thedisplay and/or uncomfortable sensation in user experience, such as jumplike movement of the hand of the avatar, a sudden change of a viewselected according to the skeleton model of the user. To reduce theundesirable artifacts and/or uncomfortable sensations, the correctioncan be applied over a period of time with increments, where eachincrement does not exceed a threshold. For example, each incrementalcorrection is limited to move the position of the hand module (117) byno more than a threshold (e.g., 5 mm); and each successive correctionsare separated by a time gap to smooth the corrections over a period oftime.

During a time period in which the optical marks (312 and 314) are withinthe field of view (319) of the camera (309) (e.g., as illustrated inFIG. 17), the computation of the rotation θ_(R) (365) that isrepresentative of the accumulated error of the arm module (115) can beperforms as illustrated in FIG. 18 and repeated periodically to performthe error reduction illustrated in FIG. 19. The time period to repeatthe calculation can be selected to balance the computation cost and themagnitude of the accumulated error. Preferably, the correction iscomputed during time periods where the optical marks (312 and 314) aremoving in the field of view (319) no faster than a threshold speed andthe positions/directions of the optical marks (312 and 314) can beaccurately determined from the images of the camera (309).

FIGS. 18 and 19 illustrate an example where the left and right arms (103and 105) remain straight. When the left and right arms (103 and 105)bend at respective elbow joints, the direction (e.g., V_(L) (373)) fromthe shoulder joint (e.g., 102) to the hand module (e.g., 119) may not bethe same as the lengthwise direction of the upper arm (105) asdetermined by the arm module (e.g., 113). In general, the direction(e.g., V_(L) (373)) from the shoulder joint (e.g., 102) to the handmodule (e.g., 119) or the hand (e.g., 108) is calculated from theorientations of the upper arm (e.g., 103), the hand (e.g., 108) and/orthe forearm connecting the upper arm (e.g., 103) and the hand (e.g.,108).

Optionally, a further arm module can be attached to the forearm and usedto track the orientation of the forearm. Alternative, the orientation ofthe forearm can be estimated and/or computed from the orientation of thehand (e.g., 108), as measured by the hand module (e.g., 119), and theorientation of the upper arm (e.g., 103), as measured by the arm module(e.g., 113), as discussed below in connection with FIGS. 20 and 21.

FIGS. 20 and 21 illustrate the computation of the orientation of aforearm (383) based on the orientations of a hand (108) and an upper arm(103) connected by the forearm (383).

FIG. 20 illustrates a skeleton model of an arm having an elbow joint(381) and a wrist joint (385). For example, the skeleton model of FIG.20 can be used in the motion processor (145) of FIG. 4 to determine theorientation of the forearm (383) that does not have an attached sensormodule, as illustrated in FIGS. 1-3, 5, 12, 16, and 17.

FIG. 20 shows the geometrical representations of the upper arm (103),the forearm (383), and the hand (108) in relation with the elbow joint(381) and the wrist (385) relative to the shoulder (102).

Each of the upper arm (103), the forearm (383), and the hand (108) hasan orientation relative to a common reference system (e.g., the shoulder(102), a room, or a location on the Earth where the user is positioned).The orientation of the upper arm (103), the forearm (383), or the hand(108) is indicated by a local coordinate system (391, 393, or 395)aligned with the upper arm (103), the forearm (383), or the hand (108)respectively.

The orientation of the upper arm (103) and the orientation of the hand(108), as represented by the local coordinate systems (391 and 395) canbe calculated from the motion parameters measured by the IMUs in themodule (113 and 119) attached to the upper arm (103) and the hand (108).

Since the forearm (383) does not have an attached IMU for themeasurement of its orientation, the motion processor (145) uses a set ofassumed relations between the movements of the forearm (383) and thehand (108) to calculate or estimate the orientation of the forearm (383)based on the orientation of the upper arm (103) and the orientation ofthe hand (108), as further discussed below.

FIG. 21 illustrates the determination of the orientation of a forearmaccording to one embodiment.

In FIG. 21, the coordinate system X₁Y₁Z₁ represents the orientation ofthe upper arm (103), where the direction Y₁ is along the lengthwisedirection of the upper arm (103) pointing from the shoulder (102) to theelbow joint (381), as illustrated in FIG. 20. The directions X₁ and Z₁are perpendicular to the direction Y₁. The direction X₁ is parallel tothe direction from the back side of the upper arm (103) to the frontside of the upper arm (103); and the direction X₁ is parallel to thedirection from the inner side of the upper arm (103) to the outer sideof the upper arm (103).

When the arm is in a vertical direction pointing downwards with the hand(108) facing the body of the user (e.g., as illustrated in FIG. 2), thelengthwise directions Y₁, Y₂, and Y₃ of the upper arm (103), the forearm(383), and the hand (108) are aligned with the vertical directionpointing downwards. When in such a position, the inner sides of theforearm (383) and the upper arm (103) are closest to the body of theuser; and the outer sides of the forearm (383) and the upper arm (103)are away from the body of the user; the directions Z₁, Z₂, and Z₃ of theupper arm (103), the forearm (383), and the hand (108) are aligned witha direction pointing sideway to the user; and the directions Z₁, Z₂, andZ₃ of the upper arm (103), the forearm (383), and the hand (108) arealigned with a direction pointing to the front of the user.

Thus, the plane X₁Y₁ is parallel to the direction X₁ from the back sideof the upper arm (103) to the front side of the upper arm (103),parallel to the lengthwise direction Y₁ of the upper arm (103), andperpendicular to the direction Z₁ from the direction from the inner sideof the upper arm (103) to the outer side if the upper arm (103). Thedirection Z₁ coincides with an axis of the elbow joint about which theforearm (383) can rotate to form an angle with the upper arm (103)between their lengthwise directions. When the upper arm (103) isextended in the sideway of the user and in a horizontal position, thedirections X₁ and Z₁ are aligned with (in parallel with) the frontdirection and vertical direction respectively.

The direction Y₂ is aligned with the lengthwise direction of the forearm(383) pointing from the elbow joint (381) to the wrist (385).

The direction Y₃ is aligned with the lengthwise direction of the hand(108) pointing from the wrist (385) towards the fingers.

When the upper arm (103) is extended in the sideway of the user and in ahorizontal position, the directions Y₁, Y₂, and Y₃ coincide with thehorizontal direction pointing the sideway of the user.

When the hand (108) is moved to an orientation illustrated in FIG. 21,the hand (108) can be considered to have moved from the orientation ofthe coordinate system X₁Y₁Z₁ through rotating (405) by an angle γ alongthe lengthwise direction Y₁ and then rotating (401) along the shortestarc (401) such that its lengthwise direction Y₃ arrives at the directionillustrated in FIG. 21. The rotation (401) along the shortest arc (401)corresponding to a rotation of the direction Y₁ by an angle β in a planecontaining both the directions Y₁ and Y₃ along an axis perpendicular toboth the directions Y₁ and Y₃ (i.e., the axis is perpendicular to theplane containing both the directions Y₁ and Y₃).

The projection of the direction Y₃ in the plane X₁Y₁ is assumed to be inthe direction of the lengthwise direction Y₂ of the forearm (383). Theprojection represents a rotation (403) of the direction Y₁ by an angle αin the plane X₁Y₁ along the direction Z₁ according to the shortest arc(403).

It is assumed that the rotation (405) of the hand (108) along itslengthwise direction is a result of the same rotation of the forearm(383) along its lengthwise direction while the forearm (383) isinitially at the orientation aligned with the coordinate system X₁Y₁Z₁.Thus, when the hand (108) has an orientation illustrated in FIG. 21relative to the orientation (X₁Y₁Z₁) of the upper arm (103), theorientation of the forearm (383) is assumed to have moved from theorientation of the coordinate system X₁Y₁Z₁ by rotating (405) along thelengthwise direction Y₁ and then rotating (403) in the plane Y₁ alongthe direction Z₁.

Since the rotations (405, 401 and 403) can be calculated from theorientation of the hand (108) relative to the orientation of the upperarm (103) (e.g., using the orientation data measured by the IMUs of thearm module (113) and the handheld module (115)), the orientation of theforearm (383) can be calculated from the rotations (405 and 403) withoutmeasurement data from an IMU attached to the forearm (383).

After the orientations of the upper arm (103), the forearm (383) and thehand (108) are obtained, the motion processor (145) can compute thepositions of the upper arm (103), the forearm (383) and the hand (108)in a three dimensional space (relative to the shoulder (102)), whichallows the application (147) to present an arm of an avatar in a virtualreality, augmented reality, or mixed reality in accordance with themovement of the arm of the user. The positions of the upper arm (103),the forearm (383) and the hand (108) in a three dimensional space(relative to the shoulder (102)) can also be used to determine thegesture made by the user in the three dimensional space to control theapplication (147) and the direction from the shoulder joint (102) to thehand (108).

Further details and examples of determining/estimating forearmorientations can be found in U.S. patent application Ser. No.15/787,555, filed Oct. 18, 2017 and entitled “Tracking Arm Movements toGenerate Inputs for Computer Systems,” the entire disclosure of which ishereby incorporated herein by reference.

After the orientations of the upper arm (103), the forearm (383) and thehand (108) are computed from the orientation measurements of the armmodule (113) and the hand module (119), the skeleton model of the arm ofFIG. 20 can be used to compute the straight line direction V_(L) (373)from the shoulder joint (102) to the hand module (119) held in the hand(108) of the user, as well as the straight line distance between theshoulder joint (102) and the hand module (119). Thus, the accumulatederror (e.g., θ_(L) (366)) can be identified in a way as illustrated inFIG. 18.

When an optical mark configured on an upper arm (103) is in the field ofview (319) of the camera (309), the direction of the optical markrelative to the shoulder (102) as observed by the camera (309) can beused to correct the orientation of the measurement of the upper arm(103) by the corresponding arm module (113); and when optical marksconfigured on the upper arm (103) and the hand (108) are in the field ofview (319) of the camera (309), the orientation of the measurement ofthe hand (103) can be corrected after the correction of the orientationof the measurement of the upper arm (103).

FIG. 22 illustrates a method to correct accumulated errors in aninertial measurement unit according to one embodiment.

For example, the method of FIG. 22 can be implemented in the system ofFIG. 4 using sensor modules (111 to 119) attached to a user in a wayillustrated in FIG. 1, the skeleton model of FIG. 18, during a timeperiod in which the optical marks (312 and 314) are within the field ofview (319) of the camera (309) illustrated in FIG. 16, after the sensormodules (111 to 119) are calibrated according to the poses of FIGS. 1 to3, and/or recalibrated using the pose of FIG. 5 or 12.

The method of FIG. 22 includes: receiving (421) an measurement of theorientation of a sensor module (e.g., 113 or 115) attached to an arm(e.g., 103 or 105) of a user; measuring (423) a direction (e.g., T_(L)(363) or T_(R) (361)) of an optical mark (e.g., 312 or 314) using a headmounted device (e.g., 111) having a camera (e.g., 309); determining(425) a difference (e.g., θ_(L) (366) or θ_(R) (365)) in the direction(e.g., T_(L) (363) or T_(R) (361)) of the optical mark (e.g., 312 or314) measured using the head mounted device (e.g., 111) and a direction(e.g., V_(L) (373) or V_(R) (371)) of the optical mark (e.g., 312 or314) determined using the measurement from the sensor module (e.g., 113or 115); and computing (427) a correction to the orientation of thesensor module based on the difference (e.g., θ_(L) (366) or θ_(R)(365)).

For example, a system to implement the method of FIG. 22 may include: aplurality of sensor modules (111 to 119) having inertial measurementunits (e.g., 121 and 131) and attached to different parts of a user(e.g., head (107), hands (106 and 108), arms (105 and 103)) to measuretheir orientations; a plurality of optical marks (e.g., 312 and 314)attached to the user; a camera (309) attached to the user; and acomputing device (141) configured to correct an accumulated error in themeasurement of an inertial measurement units (e.g., 121 or 131) bydetecting the optical marks (e.g., 312 and 314) in an image generated bythe camera (309) and identifying mismatches (e.g., θ_(L) (366) and θ_(R)(365)) in directions (e.g., T_(L) (363) and T_(R) (361)) of the opticalmarks (312 and 314) (or the sensors (e.g., 117 and 119), or parts (e.g.,106 and 108) of the users) as measured and/or calculated based on theimage generated by the camera (309) and the corresponding directions(e.g., V_(L) (373) and V_(R) (371)) of the optical marks (312 and 314)(or the sensors (e.g., 117 and 119), or parts (e.g., 106 and 108) of theusers) as measured and/or calculated from the orientation measurementsgenerated by the sensor modules (113, 115, 117 and/or 119).

For example, the computing device (141) is configured via computerinstructions to: receive orientation measurements from the sensormodules (111 to 119); receive an image captured by the camera (309); anddetect the plurality of optical marks (312 and 314) in the imagecaptured by the camera (309) to identify an accumulated error of atleast one of the received orientation measurements based on the imageand the orientation measurements. The at least one of the orientationmeasurements is corrected at least in part by removing at least aportion of the accumulated error.

For example, the accumulated error is divided into a plurality ofportions for corrections made over a period of time. Subsequentcorrections are separated by a time interval such that the correction ofthe accumulated error is smoothed out over the period of time to avoidan instantaneous correction of a significant gap. For example, each ofthe corrections for a corresponding one of portions is selected suchthat the correction moves a portion of the skeleton model of the user inspace by less than a threshold distance (e.g., 5 mm).

For example, the camera (309) is attached to or integrated in a headmounted display that may also include the computing device (141) and ahead sensor module (111) that determines the orientation of the head(107)/camera (309) of the user. The directions (e.g., T_(L) (363) andT_(R) (361)) of the optical marks (312 and 314) (or the sensors (e.g.,117 and 119), or parts (e.g., 106 and 108) of the users) are as measuredand/or calculated based on the image generated by the camera (309) andthe orientation measurement of the head module (111).

For example, at least some of the optical marks (312 and 314) areconfigured on the hand modules (117 and 119) such that the positions ofthe optical marks (312 and 314) identify the positions of the handmodules (117 and 119) and/or the positions of the hands (106 and 108) ofthe user. Alternatively, the optical marks (312 and 314) may beseparately configured on the hands (106 and 108) of the user withoutbeing on the hand modules (117 and 119). Further, some of the opticalmarks in the system are configured on the arm modules (113 and 115).Alternatively, or in combination, some of the optical marks in thesystem are attached to the user separately from the sensor modules(e.g., 111 to 119). For example, an armband or wristband having one ormore optical marks can be worn on the wrist (385) of the user or theforearm (383) without a sensor module.

In some configurations of the system, the orientation of the forearm(383) is estimated or calculated from the orientation of the upper arm(103) and the hand (108), without a sensor module being attached to theforearm (383). Alternatively, the orientation of the forearm (383) canbe measured using an additional sensor module attached to the forearm(383).

For example, when an optical mark (314) indicative of the position ofthe hand (106) of the user is in the image captured by the camera (309),the direction T_(R) (361) of the optical mark (314) (or the hand (106)or the hand module (117)) is calculated from the line of sight directionof the optical mark (314) as indicated in the image. The correspondingdirection V_(R) (371) is measured and/or calculated based on theorientation measurements of the arm module (115) and/or the hand module(117). A rotation θ_(R) (365) between the direction T_(R) (361) and thecorresponding direction V_(R) (371) identifies an accumulated error,which can be corrected by rotating the orientation measurements of thearm module (115) and/or the hand module (117) to close the gap.

For example, when both optical marks (312, 314) indicative of thepositions of the hands (106 and 108) of the user are in the imagecaptured by the camera (309), the direction T_(R) (361) of the opticalmark (314) and the direction T_(L) (363) of the optical mark (312) arecalculated respectively from the line of sight directions of the opticalmarks (312 and 314) as indicated in the image. The correspondingdirections V_(R) (371) and V_(L) (373) are measured and/or calculatedbased on the orientation measurements of the arm modules (115 and 113)and/or the hand modules (117 and 119). The rotation θ_(R) (365) betweenthe direction T_(R) (361) and the corresponding direction V_(R) (371)and the rotation θ_(L) (366) between the direction T_(L) (363) and thecorresponding direction V_(L) (373) identify accumulated errors, whichcan be corrected by rotating the orientation measurements of the armmodules (115 and 113), the hand modules (117 and 119), and/or the headmodule (111) to close the gaps (θ_(L) (366) and θ_(R) (365)).

For example, the orientation measurements of the left arm module (113)and/or left hand module (119) can be corrected according to the rotationθ_(L) (366) without considering the rotation θ_(R) (365); and theorientation measurements of the right arm module (115) and/or right handmodule (117) can be corrected according to the rotation θ_(R) (365)without considering the rotation θ_(L) (366).

Alternative, the orientation measurement of the head module (111) iscorrected based on both rotations θ_(R) (365) and θ_(L) (366) (e.g., acommon mode of θ_(R) (365) and θ_(L) (366)). In such a configuration,the orientation measurements of the left arm module (113) and/or lefthand module (119) can be corrected according to the rotation θ_(L)(366), in view of the rotation θ_(R) (365) (e.g., correcting accordingto the difference between the rotation θ_(L) (366) and the common mode);and the orientation measurements of the right arm module (115) and/orright hand module (117) can be corrected according to the rotation θ_(R)(365), in view of the rotation θ_(L) (366) (e.g., correcting accordingto the difference between the rotation θ_(R) (365) and the common mode).

The present disclosure includes methods and apparatuses which performthese methods, including data processing systems which perform thesemethods, and computer readable media containing instructions which whenexecuted on data processing systems cause the systems to perform thesemethods.

For example, the computing device (141), the handheld devices (117 and119), the arm modules (113, 115), and/or the head module (111) can beimplemented using one or more data processing systems.

A typical data processing system may include includes an inter-connect(e.g., bus and system core logic), which interconnects amicroprocessor(s) and memory. The microprocessor is typically coupled tocache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A system, comprising: a plurality of sensormodules, each having an inertial measurement unit and attached to aportion of the user to generate motion data identifying an orientationof the portion of the user; a plurality of optical marks attached to theuser; a camera attached to the user; a computing device coupled to thesensor modules and configured to: receive orientation measurements fromthe sensor modules; receive an image captured by the camera; detect theplurality of optical marks in the image captured by the camera; identifyan accumulated error of at least one of the orientation measurementsbased on the image and the orientation measurements; and correct the atleast one of the orientation measurements by removing at least a portionof the accumulated error.
 2. The system of claim 1, wherein theplurality of optical marks includes a first optical mark; and thecomputing device is further configured to: calculate a first directionof a first optical mark relative to the user based on the image;calculate a second direction of the first optical mark relative to theuser based on a skeleton model of the user and the orientationmeasurements received from the sensor modules; and compute a rotationbetween the first direction and the second direction to identify theaccumulated error.
 3. The system of claim 2, wherein the camera isattached to a head of the user.
 4. The system of claim 3, wherein thecamera is attached to the head of the user via a first sensor module inthe plurality of sensor modules; and the first direction is determinedfurther based on an orientation measurement of the first sensor.
 5. Thesystem of claim 4, wherein a second sensor module is in the plurality ofsensor modules attached to an upper arm of the user; wherein the seconddirection is determined based on an orientation measurement of thesecond sensor.
 6. The system of claim 5, wherein the first optical markis configured on a third sensor module in the plurality of sensormodules, the third sensor module attached to a hand of the user.
 7. Thesystem of claim 6, wherein the computing device is further configuredto: calculate an orientation of a forearm connected between the hand andthe upper arm based on the orientation measurement of the second sensorattached to the upper arm and the orientation measurement of the thirdsensor attached to the hand, wherein the second direction is computedbased on the orientation of the forearm.
 8. The system of claim 7,wherein the orientation of the forearm is calculated without a sensormodule attached to the forearm.
 9. The system of claim 1, wherein thecomputing device is further configured to: divide the accumulated errorinto a plurality of portions; and perform a plurality of corrections forthe plurality of portions respectively over a period of time.
 10. Thesystem of claim 9, wherein each of the plurality of corrections isselected to move a skeleton model of the user less than a thresholddistance.
 11. The system of claim 1, wherein the plurality of opticalmarks include a first optical mark attached to a left hand of the userand a second optical mark attached to a right hand of the user; and thecomputing device is further configured to: calculate a first rotationfrom a direction of the left hand determined from the first optical markin the image and the direction of the left hand determined from theorientation measurements; calculate a second rotation from a directionof the right hand determined from the second optical mark in the imageand the direction of the right hand determined from the orientationmeasurements; and determine an accumulated error in an orientationmeasurement of a sensor module on which the camera is installed based onthe first rotation and the second rotation.
 12. The system of claim 11,wherein the sensor module, the camera and the computing device areconfigured on a head mounted display device.
 13. The system of claim 11,wherein the direction of the left hand is relative to a left shoulderjoint of the user; and the direction of the right hand is relative tothe right shoulder joint of the user.
 14. The system of claim 11,wherein the plurality of sensor modules includes a first sensor moduleattached to a left upper arm of the user and a second sensor moduleattached to a right upper arm of the user; and the computing device isfurther configured to: determine an accumulated error in an orientationmeasurement of the first sensor module based at least in part on thefirst rotation; and determine an accumulated error in an orientationmeasurement of the second sensor module based at least in part on thesecond rotation.
 15. The system of claim 11, wherein the accumulatederror in the orientation measurement of the first sensor module isdetermined further based on the second rotation; and the accumulatederror in the orientation measurement of the second sensor module isdetermined further based on the first rotation.
 16. The system of claim1, wherein each of the plurality of sensor modules further includes acommunication device for a communication link with the computing device.17. The system of claim 1, wherein the inertial measurement unitincludes a micro-electromechanical system (MEMS) gyroscope.
 18. Thesystem of claim 17, wherein the inertial measurement unit furtherincludes a magnetometer and a MEMS accelerometer.
 19. A method,comprising: receiving, in a computing device, orientation measurementsfrom a plurality of sensor modules, each of the sensor modules having aninertial measurement unit and attached to a portion of the user toidentify an orientation of the portion of the user; receiving, in thecomputing device, an image captured by a camera attached to the user;detecting, by the computing device, a plurality of optical marks in theimage captured by the camera, the optical marks attached to the user;identifying an accumulated error of at least one of the orientationmeasurements based on the image and the orientation measurements; andcorrecting the at least one of the orientation measurements by removingat least a portion of the accumulated error.
 20. A non-transitorycomputer storage medium storing instructions which, when executed by acomputing device, instructs the computing device to perform a method,the method comprising: receiving, in the computing device, orientationmeasurements from a plurality of sensor modules, each of the sensormodules having an inertial measurement unit and attached to a portion ofthe user to identify an orientation of the portion of the user;receiving, in the computing device, an image captured by a cameraattached to the user; detecting, by the computing device, a plurality ofoptical marks in the image captured by the camera, the optical marksattached to the user; identifying an accumulated error of at least oneof the orientation measurements based on the image and the orientationmeasurements; and correcting the at least one of the orientationmeasurements by removing at least a portion of the accumulated error.